New benchmark derived from otel-launcher-go #3395

jmacd · 2022-10-25T17:09:10Z

This is a near-identical copy of the benchmark used in developing the Lighstep metrics SDK.

This measures end-to-end performance of the synchronous instrument implementation using Counters. This should help settle the question in #3199.

The current numbers:

goos: darwin
goarch: arm64
pkg: go.opentelemetry.io/otel/sdk/metric
BenchmarkCounterAddNoAttrs-10                 	51228051	        22.61 ns/op	       0 B/op	       0 allocs/op
BenchmarkCounterAddOneAttr-10                 	 7272368	       164.0 ns/op	     152 B/op	       3 allocs/op
BenchmarkCounterAddOneInvalidAttr-10          	 4732094	       254.3 ns/op	     280 B/op	       3 allocs/op
BenchmarkCounterAddManyAttrs-10               	 2170496	       638.3 ns/op	     301 B/op	       3 allocs/op
BenchmarkCounterAddManyInvalidAttrs-10        	 1676473	       658.9 ns/op	     378 B/op	       3 allocs/op
BenchmarkCounterAddManyFilteredAttrs-10       	 1610043	       676.1 ns/op	     381 B/op	       3 allocs/op
BenchmarkCounterCollectOneAttr-10             	 2484734	       475.9 ns/op	     456 B/op	       8 allocs/op
BenchmarkCounterCollectTenAttrs-10            	  507835	      2352 ns/op	    2512 B/op	      35 allocs/op
BenchmarkCounterCollectTenAttrsTenTimes-10    	   51271	     23737 ns/op	   25120 B/op	     350 allocs/op

codecov · 2022-10-25T17:14:13Z

Codecov Report

Merging #3395 (fea2e28) into main (c8a13d6) will decrease coverage by 0.2%.
The diff coverage is 61.8%.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #3395     +/-   ##
=======================================
- Coverage   77.9%   77.7%   -0.3%     
=======================================
  Files        164     164             
  Lines      11361   11457     +96     
=======================================
+ Hits        8861    8909     +48     
- Misses      2301    2346     +45     
- Partials     199     202      +3

Impacted Files	Coverage Δ
exporters/otlp/otlpmetric/otlpmetricgrpc/config.go	`41.8% <0.0%> (-3.3%)`	⬇️
exporters/otlp/otlpmetric/otlpmetrichttp/config.go	`85.7% <0.0%> (-14.3%)`	⬇️
sdk/metric/aggregation/aggregation.go	`77.4% <ø> (ø)`
sdk/metric/reader.go	`100.0% <ø> (+24.4%)`	⬆️
exporters/otlp/otlpmetric/exporter.go	`67.2% <25.0%> (-17.8%)`	⬇️
exporters/otlp/otlpmetric/otlpmetricgrpc/client.go	`84.9% <42.8%> (-3.0%)`	⬇️
exporters/otlp/otlpmetric/otlpmetrichttp/client.go	`77.5% <42.8%> (-1.6%)`	⬇️
sdk/metric/manual_reader.go	`78.6% <65.2%> (-6.0%)`	⬇️
...xporters/otlp/otlpmetric/internal/oconf/options.go	`79.8% <72.4%> (-1.7%)`	⬇️
exporters/stdout/stdoutmetric/config.go	`72.9% <72.4%> (-0.8%)`	⬇️
... and 8 more

jmacd · 2022-10-25T17:16:46Z

Although these benchmarks allow comparing the two SDKs, that wasn't my goal. Here is the corresponding output for otel-launcher-go:

goos: darwin
goarch: arm64
pkg: github.com/lightstep/otel-launcher-go/lightstep/sdk/metric
BenchmarkCounterAddNoAttrs-10                 	30648045	        37.15 ns/op	       0 B/op	       0 allocs/op
BenchmarkCounterAddOneAttr-10                 	12716595	        93.52 ns/op	      64 B/op	       1 allocs/op
BenchmarkCounterAddOneInvalidAttr-10          	 8578038	       140.8 ns/op	     128 B/op	       1 allocs/op
BenchmarkCounterAddManyAttrs-10               	 1000000	      1157 ns/op	     569 B/op	       6 allocs/op
BenchmarkCounterAddManyInvalidAttrs-10        	  743876	      1799 ns/op	    1092 B/op	      10 allocs/op
BenchmarkCounterAddManyFilteredAttrs-10       	 1000000	      1426 ns/op	     954 B/op	       8 allocs/op
BenchmarkCounterCollectOneAttrNoReuse-10      	 2412259	       493.3 ns/op	     400 B/op	       7 allocs/op
BenchmarkCounterCollectOneAttrWithReuse-10    	 3470206	       345.5 ns/op	     136 B/op	       3 allocs/op
BenchmarkCounterCollectTenAttrs-10            	  663994	      1762 ns/op	     712 B/op	      12 allocs/op
BenchmarkCounterCollectTenAttrsTenTimes-10    	   68974	     17650 ns/op	    7129 B/op	     120 allocs/op

See it uses fewer allocations and is somewhat faster when the attribute set is re-used (due to "fingerprinting"), otherwise is somewhat slower due to extra complexity associated with memory management.

jmacd · 2022-10-25T17:18:06Z

Also note @bogdandrutu I could not identify a performance impact of the following diff

diff --git a/sdk/metric/view/view.go b/sdk/metric/view/view.go
index c6f20018..963c7240 100644
--- a/sdk/metric/view/view.go
+++ b/sdk/metric/view/view.go
@@ -71,7 +71,7 @@ func New(opts ...Option) (View, error) {
 
 // TransformInstrument will check if an instrument matches this view
 // and will convert it if it does.
-func (v View) TransformInstrument(inst Instrument) (transformed Instrument, match bool) {
+func (v *View) TransformInstrument(inst Instrument) (transformed Instrument, match bool) {
 	if !v.match(inst) {
 		return Instrument{}, false
 	}
@@ -89,7 +89,7 @@ func (v View) TransformInstrument(inst Instrument) (transformed Instrument, matc
 
 // AttributeFilter returns a function that returns only attributes specified by
 // WithFilterAttributes. If no filter was provided nil is returned.
-func (v View) AttributeFilter() func(attribute.Set) attribute.Set {
+func (v *View) AttributeFilter() func(attribute.Set) attribute.Set {
 	if v.filter == nil {
 		return nil
 	}
@@ -99,27 +99,27 @@ func (v View) AttributeFilter() func(attribute.Set) attribute.Set {
 	}
 }
 
-func (v View) matchName(name string) bool {
+func (v *View) matchName(name string) bool {
 	return v.instrumentName == nil || v.instrumentName.MatchString(name)
 }
 
-func (v View) matchScopeName(name string) bool {
+func (v *View) matchScopeName(name string) bool {
 	return v.scope.Name == "" || name == v.scope.Name
 }
 
-func (v View) matchScopeVersion(version string) bool {
+func (v *View) matchScopeVersion(version string) bool {
 	return v.scope.Version == "" || version == v.scope.Version
 }
 
-func (v View) matchScopeSchemaURL(schemaURL string) bool {
+func (v *View) matchScopeSchemaURL(schemaURL string) bool {
 	return v.scope.SchemaURL == "" || schemaURL == v.scope.SchemaURL
 }
 
-func (v View) matchInstrumentKind(kind InstrumentKind) bool {
+func (v *View) matchInstrumentKind(kind InstrumentKind) bool {
 	return v.instrumentKind == undefinedInstrument || kind == v.instrumentKind
 }
 
-func (v View) match(i Instrument) bool {
+func (v *View) match(i Instrument) bool {
 	return v.matchName(i.Name) &&
 		v.matchScopeName(i.Scope.Name) &&
 		v.matchScopeSchemaURL(i.Scope.SchemaURL) &&

I believe this settles the question in #3199: it's no big deal to use structs for Views. If there's a statistical difference here, it's slight.

hanyuancheung

👍

sdk/metric/benchmark_test.go

…to jmacd/newbench

…o jmacd/newbench

sdk/metric/view/view.go

Joshua MacDonald added 2 commits October 25, 2022 08:42

New benchmark derived from otel-launcher-go

b350bab

edit

4fb4d64

jmacd requested review from MrAlias, Aneurysm9, evantorrie, XSAM, dashpole, MadVikingGod, pellared, hanyuancheung and dmathieu as code owners October 25, 2022 17:09

jmacd added the Skip Changelog PRs that do not require a CHANGELOG.md entry label Oct 25, 2022

Merge branch 'main' into jmacd/newbench

d1f7b6a

dmathieu approved these changes Oct 26, 2022

View reviewed changes

hanyuancheung approved these changes Oct 26, 2022

View reviewed changes

MrAlias reviewed Oct 27, 2022

View reviewed changes

sdk/metric/benchmark_test.go Outdated Show resolved Hide resolved

sdk/metric/benchmark_test.go Outdated Show resolved Hide resolved

sdk/metric/benchmark_test.go Show resolved Hide resolved

sdk/metric/benchmark_test.go Outdated Show resolved Hide resolved

Joshua MacDonald added 5 commits October 31, 2022 15:35

refactoring from PR feedback

476d044

Merge branch 'jmacd/newbench' of github.com:jmacd/opentelemetry-go in…

5a51fb5

…to jmacd/newbench

Merge branch 'main' of github.com:open-telemetry/opentelemetry-go int…

23ac68e

…o jmacd/newbench

refactoring from PR feedback

71b6ab8

update for upstream main

a295f28

dashpole reviewed Nov 1, 2022

View reviewed changes

sdk/metric/view/view.go Outdated Show resolved Hide resolved

revert view change

6a2d5bc

dashpole approved these changes Nov 1, 2022

View reviewed changes

MrAlias approved these changes Nov 1, 2022

View reviewed changes

Merge branch 'main' into jmacd/newbench

fea2e28

MrAlias merged commit 49b62ae into open-telemetry:main Nov 1, 2022

MrAlias added this to the Metric v0.34.0 milestone Dec 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New benchmark derived from otel-launcher-go #3395

New benchmark derived from otel-launcher-go #3395

jmacd commented Oct 25, 2022

codecov bot commented Oct 25, 2022 •

edited

Loading

jmacd commented Oct 25, 2022

jmacd commented Oct 25, 2022

hanyuancheung left a comment

New benchmark derived from otel-launcher-go #3395

New benchmark derived from otel-launcher-go #3395

Conversation

jmacd commented Oct 25, 2022

codecov bot commented Oct 25, 2022 • edited Loading

Codecov Report

jmacd commented Oct 25, 2022

jmacd commented Oct 25, 2022

hanyuancheung left a comment

Choose a reason for hiding this comment

codecov bot commented Oct 25, 2022 •

edited

Loading